Introduction - DataFrames & Spark SQL

Schema Definition & Metadata

Defining explicit schemas with StructType/StructField and inspecting DataFrame structure via printSchema, describe, and summary.

Core DSL Transformations

Selecting, filtering, modifying, sorting, and grouping DataFrames using select(), filter(), withColumn(), orderBy(), and groupBy().

Joins, Sets & Null Handling

Merging datasets with joins and unions, deduplicating with distinct(), and imputing or dropping null values.

Advanced Ops & Spark SQL

Tuning partitions, computing window functions, exploding nested types, and querying DataFrames with registered SQL views.

DSL transformations to select columns, calculate formulas, and project SQL expressions with selectExpr().

Restricting row results in a DataFrame based on boolean logical column expressions and string filter queries.

Programmatically adding, renaming, dropping, and casting column structures via withColumn().

Sorting rows in a DataFrame based on single or multiple fields in ascending or descending orders.

Summarizing grouped datasets in PySpark using aggregate functions like sum, avg, count, min, max.

Merging different datasets using inner, outer, left, right, left-semi, and left-anti joins.

Concatenating datasets vertically using unsafe union() vs. name-aligned safe unionByName().

Discarding duplicate row records using distinct() and column-specific dropDuplicates().

Imputing, filling, replacing, and dropping null values using dropna, fillna, and replace.

Tuning distributed parallelism and data layouts using repartition() vs. node-local coalesce().

Inspecting DataFrame structures and generating statistical descriptions using printSchema, describe, and summary.

Calculating rolling averages, ranks, lags, and lead analytics using Window structures.

Ingesting, parsing, and exploding structs, arrays, and JSON objects using explode and from_json.

Hands-on primer combining select(), filter(), and withColumn() DSL syntax before the individual deep-dives.

Using groupBy() and agg() to run single and multi-aggregate computations over grouped data.

Defining explicit schemas programmatically using StructType and StructField to ensure type safety.

Registering local or global temporary views to run standard ANSI SQL queries via spark.sql().

Tracing data metrics through aggregation filters on a Fire Calls dataset using DataFrame operations.

Scenario questions covering schema inference vs. explicit schemas, group-by internals, and type safety constraints.